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(57) Abstract 

A control system (20) for automatically detecting moisture on the windshield (26) of a vehicle. The automatic moisture detecting 
system (20) includes an optical system (30) for imaging a portion of the windshield (26) onto an image array sensor (32), such as a CMOS 
active pixel sensor. The voltages of each of the pixels which represents the illumination level is converted to a corresponding gray scale 
value by an analog digital converter (35). The gray scale values corresponding to the image are stored in memory (38). The spatial 
frequency composition of the gray scale values are analyzed to determine the amount of rain present. In order to provide a control signal 
to control the operation of the windshield wipers (40) of the vehicle as a function of the amount of moisture present. The system is also 
adapted to detect the level of fog both on the interior of the windshield (26) as well as the exterior of the windshield (26). By providing a 
system for automatically detecting the presence of fog on the interior and exterior of the windshield (26), serious performance limitations 
of known automatic rain sensors are eliminated. f 
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MOISTURE SENSOR AND WINBSEUELD FOG DETECTOR 

Technical Fielfl 

The present invention relates to a system for automatically detecting 
the presence of moisture on a surface such as the surface of a windshield of a 
5 vehicle for an automobile, in order to automatically actuate the vehicle's 
windshield wipers and/or defroster or defogging system. 

Descriptio n of the Prior Art 

In conventional windshield wiper systems, the windshield wipers are 
actuated based upon the elapsed time between wipes rather than the moisture level 

10 on the exterior of the windshield. During conditions of relatively consistent 
rainfall, for example, the time interval can be adjusted to correspond to the amount 
of time in which the rain accumulates to the point of the desired visibility level. 
Unfortunately the rate of rainfall may vary dramatically over a given period of 
time. Additionally, traffic conditions may also cause varying amounts of rain to 

15 fall on the windshields due to traffic conditions, such as a passing truck or the like. 
As a result, during such conditions, the wiper time interval must frequently be 
adjusted which can be cumbersome. 

Various systems are known which automatically control the interval 
between wipes of the windshield wipers based upon moisture on the vehicle 

20 windshield. In some known systems, various coatings are applied to the vehicle 
windshield. Electrical measurement of those coatings are used to provide an 
indication of the moisture content on the windshield. Unfortunately such methods 
require relatively expensive processes which makes such system commercially non- 
viable. Other systems for automatically sensing the moisture content on a vehicle 
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windshield are also known. For example, optical systems are known which 
measure the difference of reflected light of a dry windshield versus a wet 
windshield. Unfortunately, the optical method is susceptible to interference from 
external light sources and thus provides inadequate performance. Other known 
5 systems must be adhered to the windshield which complicates the windshield 
replacement. As a result of such complications, moisture sensors are rarely found 
on vehicles. 

Another system for automatically detecting the moisture content on 
a windshield is disclosed in Japanese Laid Open Patent Application No. 

10 Hei> (1995)-286130 which describes the use of a charge coupled device (CCD) 
image sensor to image a portion of the vehicle windshield in order to detect 
raindrops. The system described therein computes the sum of the differences 
between each pixel and the average of all pixels. Unfortunately headlamps of 
oncoming vehicles will create a bright spot in the image which would be difficult 

15 to completely blur and likely be interpreted as rain. Moreover, in order for such 
a system to work effectively, images from the distant scene must be completely 
blurred. Otherwise, there will be dark and light regions in the distant scene. 
Although there is no optical system disclosed in the Japanese laid open Patent 
Application for accomplishing this objective, it would be very difficult to develop 

20 an optical system to completely blur an oncoming headlamp. Failure to blur 
oncoming headlamps could cause false triggering of the system disclosed in the 
above-identified Japanese laid open Patent Application. 

Another problem with automatic rain detection systems is the 
inability of the system to detect the operation of the windshield wipers. In certain 

25 cold climate conditions, the windshield wipers are known to freeze to the 
windshield. In such a situation, since the moisture is not being removed by the 
wipers, an automatic rain sensing device would continuously command the wipers 
to actuate, even though the wipes are frozen to the windshield, potentially 
damaging the windshield wiper system. 
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Another known problem with known systems is the inability to 
detect fog on the interior and exterior of the windshields. As mentioned above, 
automatic moisture detection systems, such as disclosed in the above-identified laid 
open Japanese Patent Application, are based upon the ability to detect raindrops on 
5 the windshield. When a uniform fog or mist covers the vehicle windshield, 
systems, such as the system disclosed in the Laid Open Japanese Patent 
Application, are unable to sense such moisture on the exterior of the windshield. 
As a result, during such a condition, the windshield wipers will have to be 
manually actuated, thereby partially defeating the purpose of an automatic rain 
10 sensor and windshield wiper control system making the feature a lot less desirable. 

In other situations, fog develops on the inside of the windshield 
independent of the moisture content on the exterior of the windshield. In such a 
condition automatic rain sensing systems, such as disclosed in the Laid Open 
Japanese Patent Application, are unable to detect the moisture content on the 
exterior of the vehicle windshield until after the fog on the interior of the 
windshield is cleared. In such a condition, a defroster or defogger system would 
have to be manually actuated to remove the interior fog on the windshield. The 
automatic rain sensor would not be operable during such a condition until the fog 
on the interior of the windshield is sufficiently cleared. 

The desirability of having automatic rain sensing is to have a system 
which automatically controls the windshield wipers during typical climatic 
conditions, such as rain, snow and fog. When the wiper system has to be operated 
manually during such typical conditions such a feature becomes undesirable. 

Summary Off The InYentipn 

It is an object of the present invention to provide a system which 
solves the problems of the prior art. 
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It is yet another object of the present invention to provide a system 
for automatically detecting the moisture content on the windshield of a vehicle. 

It is yet another object of the present invention to provide a system 
for automatically detecting moisture on the windshield of a vehicle during common 
5 climatic conditions, such as rain, snow and fog. 

It is yet a further object of the present invention to provide a system 
for automatically detecting the presence of fog on the exterior of a windshield. 

It is yet another object of the present invention to provide a system 
for automatically detecting the presence of fog on the interior of a windshield. 
10 It is yet a further object of the present invention to provide a system 

for automatically sensing the windshield wipers crossing a portion of the 
windshield. 

Briefly the present invention relates to a system for automatically 
detecting moisture on the windshield of a vehicle. The automatic moisture 

15 detecting system includes an optical system for imaging a portion of the windshield 
onto an image array sensor, such as a CMOS active pixel sensor. The voltage of 
each of the pixels, which represents the illumination level, is converted to a 
corresponding gray scale value by an analog to digital converter. The gray scale 
values corresponding to the image are stored in memory. The spatial frequency 

20 composition of the gray scale values are analyzed to determine the amount of rain 
present in order to provide a control signal to control the operation of the 
windshield wipers of the vehicle as a function of the amount of moisture present. 
The system is also adapted to detect fog both on the interior of the windshield as 
well as the exterior of the windshield. By providing a system for automatically 

25 detecting the presence of fog on the interior and exterior of the windshield, serious 
performance limitations of known automatic rain sensors during typical climatic 
conditions are eliminated. 



WO 99/14088 



PCT/US98/18899 



-5- 

Brief Description* Of The Drawing 

These and other objects of the present invention will be readily 
understood with reference to the following specification and attached drawing 
wherein: 

5 FIG. 1 is a physical diagram which illustrates a vehicle windshield 

and an attached rearview mirror illustrating a system in accordance with the 
present invention. 

FIG. 2 is an enlarged view of a portion of the system for sensing 
moisture on the exterior of a windshield in accordance with the present invention. 
10 FIG. 3 is a physical diagram of a system in accordance with an 

alternate embodiment of the invention for sensing fog which illustrates the 
projection of a beam of light onto the windshield for fog detection. 

FIGs. 4a and 4b are computer simulated spot diagrams which 
illustrate the performance of the optical system in accordance with the present 
15 invention during moisture and non-moisture conditions, respectively. 

FIG. 5 is a flow diagram for the system in accordance with the 
present invention. 

FIG. 6 is a block diagram of the system in accordance with the 
present invention. 

20 

Detailed! Description Of The Invention 

An automatic moisture sensing system in accordance with the 
present invention is able to detect moisture on the windshield of a vehicle in order 
to automatically control the vehicle's windshield wiper, defroster and/or defogging 
25 systems. The system for automatically sensing moisture on a vehicle windshield 
eliminates many of the performance deficiencies of known automatic moisture 

t 

i 
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sensing systems at a commercially viable cost. As used herein, the term moisture 

i 

is used to designate various types of moisture and precipitation which can be found 
on the windshield of a vehicle during various climatic conditions, such as rainfall, 

i 
\ 

snow fall, ice, fog as well as other substances that are commonly deposited on a 

vehicle windshield such as bugs, dust and the like. The system is able to provide 

! 

superior performance to other known systems during rather common climatic 
conditions, such as ice, fog and varying levels of rain and snow fall and the like. 

As will be discussed in more detail below, a portion of the 

S 

windshield is imaged onto an image array sensor. An optical system which forms 
a portion of the present invention causes raindrops and other sources of moisture 
on the windshield to be sharply focused while distant objects beyond the 
windshield are severely blurred in the image. The processing system analyzes the 
image for sharp discontinuities caused by the edges of the water rain droplets or 
other moisture and by random focusing of the distant objects by the droplets. 
These discontinuities represent high spatial frequency components . The magnitude 
of high spatial frequency components is a measure of the amount of the rain or 
other moisture on the vehicle which can be used to automatically control the 
vehicle windshield wipers. In an alternate embodiment of the invention the system 

is adapted to sense fog on the interior and exterior of the windshield in order to 

I 

prevent spurious operation of the automatic moisture sensing system. As such, the 
present invention eliminates many of the various performance limitations of known 
automatic moisture sensing systems. 

In yet another alternate embodiment of the present invention, the 
system is able to detect operation of the windshield wipers in order to prevent 
spurious operation and damage to the windshield wiper system during conditions 
when the windshield wipers are stuck or frozen to the windshield. As discussed 
above, the system analyzes the image of a portion of the windshield for sharp 
discontinuities which will have relatively high spatial frequency components. The 
magnitude of these high spatial frequency components is used to represent the 
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measure of moisture or other substances on the windshield. Thus, dust, bugs and 
other substances that will initially be treated as moisture. However, as discussed 
above, the system has the ability to automatically sense an operation of the 
windshield wiper blades. Thus, if the substance, which may be ice, dirt, cracks 
5 or other substances not removable by the windshield wipers; remains on the 
windshield after one or more wipes, the system in accordance with the present 
invention may be configured to ignore such substances in order to prevent further 
spurious operation of the vehicle windshield wiper system. 

Referring to FIG. 1, the automatic moisture sensing system in 
10 accordance with the present invention generally identified with the reference 
numeral 20. The automatic moisture sensing system may be mounted stationary 
in the mounting bracket 22 of an automobile rearview mirror 24 or alternatively 
mounted in the rear portion of the rearview mirror housing 24. The automatic 
moisture sensing system 20 includes an image sensor mounted, for example, 2-3 
inches behind the vehicle windshield 26 with the optical axis substantially parallel 
to ground or slightly angled to the ground. The angle of the windshield 26 in a 
modern passenger car is about 27° . Such a configuration may cause the raindrops 
and other moisture to be at a different distance from the image sensor depending 
on where the moisture is with respect to the field of view of the image sensor. To 
help compensate for this problem, the image sensor may be angled approximately 
10° toward the windshield 26 such that the top of the sensor 20 is moved closer to 
the windshield 26. 

There are four main components to the automatic moisture sensing 
system 20: an imaging optical system; a light emitting diode; an image sensor; and 
a processor. The imaging optical system is shown best in FIG. 2 while the image 
sensor is illustrated in FIG. 2 and in FIG. 6. A flow diagram for the 
microcontroller is illustrated in FIG. 5. 
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IrmmW Optical /System 

The imaging optical system, generally identified with the reference 
numeral 30 (FIG. 2), is used to image a predetermined portion of the windshield 
26 onto an image sensor 32 such that objects at the approximate distance of the 
5 windshield 26 are sharply in focus at the image plane while objects at a longer 
distance are out of focus and blurred. The area of the windshield 26 that is imaged 
must be large enough that the probability of receiving raindrops during relatively 
light rain conditions is significant. Moreover, the imaged area of the windshield 
must also be in the area of the windshield that is wiped by the windshield wipers. 

10 The imaging optical system may include a single biconvex lens 33, 

used as an imaging lens. The lens 33 may have a diameter of 6mm; a front and 
rear radius of curvature of 7mm for each surface and a center thickness of 2.5mm. 
The front surface of lens 33 may be positioned 62mm from the outer surface of the 
windshield 26. The imaging lens 33 may be carried by a mechanical lens mount 

15 34 which forms a stop 36 of about 5mm diameter directly in front of the lens 33. 
The image sensor may be located about 8.55mm from the rear surface of the lens 
33 and as mentioned above, slightly angled by about 10°. 

More elaborate optical systems, for example, with multiple 
elements, aspherical elements, or defractive objects could all be used especially if 

20 shorter distance from the windshield is a desired feature. However, since the 
collected images are not for photographic purposes, such optical quality is not 
necessary in an application for automatic moisture detection. A single lens may 
also be used molded in acrylic or other clear plastics at a relatively low cost. 
Various companies including Polaroid and Kodak specialize in high performance 

25 molded plastic optics. 

FIG. 4 illustrates a computer simulation of the performance of the 
imaging system illustrated in FIG. 2. In particular, FIG. 4a is a spot diagram of 
the imaging of approximately parallel light rays from a relatively distant object on 
the optical axis onto an image plane. FIG. 4b is a spot diagram of the imaging of 
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a point on the optical axis at the distance of the outer surface of the windshield. 
Upon comparison of the spot diagrams of FIGs. 4a and 4b, it is evident that the 
optical system is able to blur light coming from distant objects while focusing light 
from objects at the windshield distance. 
5 Occasionally when driving up a hill, the vehicle could be positioned 

in such a way that the sun is directly imaged by the device. The radiative loading 
caused by this alignment may damage the image sensor 32 over time. In order to 
alleviate such a problem, an electrochromic filter may be used to temporarily 
eliminate most of the sunlight from the image plain. Other optical electronic or 
10 optical mechanical devices could also be used. 

Imaee Sensor 

The image sensor 32 may be a CMOS active pixel sensor. CMOS 
active pixel sensors are a recent breakthrough in imaging technology that allow low 
cost, high sensitivity imaging on a chip manufactured in a CMOS process. Such 

15 CMOS active pixel sensors have several advantages over other sensors including 
low power consumption, popular CMOS production techniques, low cost and 
ability to integrate additional circuitry on the same chip, variable read out windows 
and a variable light integration time. Such CMOS active pixel sensors are 
commercially available from Photobit LLC, La Cresenta, California. While 

20 CMOS active pixel sensors have substantial advantages, other image sensors are 
also suitable and are considered to be within the scope of the present invention. 
The size and number of pixels is determined to image an area of the windshield 
sufficiently large and enough detail to adequately detect light rain while remaining 
cost effective. For example, a 64 x 64 active pixel, 40^m pixel size array will 

25 image approximately a 25mm x 40mm area on a standard passenger car 
windshield. 
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Processing And Control 

A block diagram of the automatic moisture sensing circuitry is 
shown in FIG. 6. As mentioned above, a predetermined portion of the windshield 
26 is imaged onto an image array sensor 32. The analog voltage of each of the 
5 pixels within the sensor 32 is converted to digitized gray scale value by way of an 
analog to digital converter. The analog to digital converter 35 is operated under 
the control of a timing and control circuit 37 which, in turn, is controlled by a 
microcontroller 38. The timing and control circuit 37 is described in detail in U.S. 
patent application, Serial No. 933,210, entitled "Control Circuit For Image Array 

10 Sensors", by Jon Bechtel and Joseph Stam. A suitable microcontroller 38 is a 
Motorola type 68HC08XL36. However, it is commonly known that such 
microcontrollers do not contain sufficient random access memory (RAM) to store 
entire image from 50 x 50 pixel image sensor. In such a situation, a windowing 
feature of the CMOS imaging sensors may be used to alternatively image and 

15 process different regions of small enough size for the onboard RAM of the 
microcontroller 38. 

As discussed above, the system analyzes the digitized gray scale 
values for sharp edges which are representative of rain drops or other moisture by 
analyzing the spatial high frequency components. The magnitude of the spatial 

20 high frequency components is used to control a windshield wiper motor control 40 
such that the frequency of wiping of the windshield wiper blades (i.e. time interval 
between wipes) is controlled as a function of the amount of moisture on the 
windshield. As will be discussed in more detail below, the system also is able to 
detect the fog on the interior and exterior of the windshield. Thus, the 

25 microcontroller 38 may also be used to automatically control the vehicle defroster 
or defogging system 42. In order to provide selectivity of a system, a driver/on 
off sensitivity control circuit 44 may be provided. This control circuit 44 may be 
used in special circumstances for example when the vehicle is in an automatic car 
wash to prevent spurious operation of the system. 



WO 99/14088 



PCT/US98/18899 



-11- 

Once an image is acquired by the image array sensor 32, the 
luminance on each pixel, represented by an analog voltage, is converted to a digital 
gray scale value by the analog digital converter 35. These values are written to 
memory, which may be on board the microcontroller 38 and processed by the 
5 microcontroller 38 or alternatively a digital signal processor. 

Rain is detected by quantifying the discontinuity resulting from 
sharp edges of the raindrops on the windshield. These sharp edges are caused by 
the focused image of the rain or other moisture droplet along with the random 
optical imaging of far field objects by the droplets or other moisture. As discussed 

10 in "Digital Image Processing" by R.C. Gonolez and R.E. Woods, Addison-Wesely 
1992, the images may be analyzed in terms of their spatial frequency composition. 
Spatial frequency composition analysis is analogous to Fourier analysis, commonly 
used in both digital and analog signal processing. The process of taking a Fourier 
transform of a signal and determining its frequency composition can readily be 

15 applied to two dimensional signals. When the two dimensional signal is an image, 
it is common to use the term spatial frequency. The spatial frequency composition 
of an image can be evaluated using a two dimensional Fourier transform of the 
image. The transform is given by equation (1) as follows: 

0) 

P(f*y<*}= f ffixrfe - J ^dxdy 

20 where: f (x,y)is the value of the pixel in the original image located at pixel x,y; 
F (o) x , G> y ) is the value of the Fourier transform of the image at pixel location <o x , 

o> y ; j is the complex number yf^l . 

Equation (1) describes the Fourier transform for continuous infinite 
two dimensional signals. This function can be readily adapted to discrete, finite 
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two dimensional signals resulting from digital images. Applying spatial frequency 
analysis techniques, the rough edges or "roughness" of an image can be relatively 
accurately quantified. For example, a Fourier transform can be performed on a 
very blurry image. In such an analysis the value of F (o> x , G) y ) for low magnitudes 
5 <o of spatial frequencies o> x , co y will be high while the value of F (<o x , (o y ) at high 
magnitudes of co x , a> y will be low. The value of F (g> x , o) y ) where o> x , o> y are both 
0 is always the average gray scale value of the image. 

Alternatively, a Fourier analysis of a sharply focused image with 
many edges will result in the values of F (o> x , co y ) for large magnitudes of g> x , G) y 
10 being high. A digital filter can be used to select particular spatial frequency 
regions. A relatively simple implementation of such a filter for image processing 
uses a 3 x 3 matrix supplied to a 3 x 3 pixel neighborhood as illustrated below: 



A 


B 


C 


D 


E 


F 


G 


H 


I 



A new image g(x,y) can be formed which is the resultant image of applying the 
filter to the current image. The image may be processed in a looping fashion for 
every pixel with a value of f(x,y) at a location defined by the variables x and y. 
In the above matrix, the location of the coefficient E corresponds to the current 

20 pixel at x and y. The pixel at location x and y in the new image has a value given 

by equation (2) below: | 
(2) g(x,y) = A-f(x-l,y-l) + B-f(x,y-l) + C'f(x-f 1 ,y-l) + D-f(x- 1 
l,y)+E-f(x,y)+F-f(x+l,yH^ j 
A special filter commonly used is a Laplacian filter. The Laplacian 

25 is the second-order derivative of a two dimension function f(x,y) given by equation 
3: 
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dy 2 dx 2 

This Laplacian function can be implemented in discrete space using the 3 x 3 
matrix described above with the coefficients as follows: E=4; B,D,F & H = -1 
and the rest of the coefficients zero. Other coefficient combinations can also be 
5 used to compute variations of the discrete Laplacian as long as the coefficient E is 
positive and the rest are negative and that the sum of all coefficients is zero. The 
spatial frequency response of any 3X3 filter is determined by equation (4) below: 

(4) , , 

m = -\ n--\ 

where: 

10 H(co x , co y ) is the frequency response of the filter for frequencies co x 

and o) y ; the function h(m,n) describes the coefficients of the matrix above; the 
coefficient E is the value of h(0,0), A is the value of h(-l,-l), etc;y is the complex 

number y^T . 

By analyzing the frequency response of the discrete 3x3 Laplacian 
filter using equation (4), it is evident that the discrete 3x3 Laplacian is a high pass 
filter. By modifying the coefficients, the particular response of the filter can be 
adjusted. Additionally, a 5 x 5 or larger filter can be used for even finer control 
over the response. 

Rain drops and other moisture can be detected by using a 3 x 3 
Laplacian filter described above. Every pixel is examined individually in a loop 
fashion and a variable used to store the total amount of "moisture" detected. The 



15 



20 
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Laplacian for each pixel is computed using the formula for g(x,y) described above 
with the Laplacian coefficients. 

A flow diagram in accordance with the present invention is 
illustrated in FIG. 5. Initially in step 46 an image of the windshield is acquired. 
5 As mentioned above, the optical system in accordance with the present invention 
images the scene in such a way that distant objects are out of focus and objects at 
the windshield distance are in focus. Thus, if there is no moisture or other matter 
on the windshield, only a blurry image of distant objects will be captured. A 
blurry image will have a relatively low high frequency spatial component. Thus, 

10 the value of the Laplacian in such a situation will be relatively low. If there is rain 
or other moisture on the windshield, the drops will be in focus and the image will 
contain relatively large high frequency components. Despite the blurring of distant 
objects by the optical system 30, oncoming head lamps from other vehicles which 
are relatively bright may contribute a significant high frequency spatial component. 

15 In order to filter this component out, pixels which have a gray scale value above 
the saturation level of the analog to digital converter (i.e. pixels with a gray scale 
value at or near 255) may be skipped. 

In step 48, the Laplacian of each pixel is computed and the results 
are stored. More particularly, if the magnitude of a Laplacian is above a given 

20 threshold, indicating a large enough high frequency spatial component to indicate 
rain or other moisture, this value is summed with the values of the other pixels to 
indicate the total value of rain or other moisture which is compared against a 
threshold, which may be a user set threshold as indicated in step 50. If the sum 
of the Laplacian of each of the pixels is greater than the threshold as determined 

25 in step 52, windshield wipers are actuated in step 54. Otherwise the system loops 
back to step 46 and acquires a new image of the windshield. 

The threshold indicated in step 52 may be a fixed threshold or a 
variable threshold. In applications where variable threshold is used, the threshold 
may be a user set threshold implemented by a control knob or slide with a voltage 
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output. This voltage output may then be sampled and converted to a digital value 
which is appropriately scaled for comparison with the sum of the pixels which 
indicate the total amount of rain. 

In order to prevent spurious operation of the system, the operation 
5 of the wiper blade is sensed in step 56. In particular, a small sub window of the 
image array sensor may be selected to allow more cycles to be processed per 
second in order to detect a relatively rapidly moving windshield wiper. An image 
of the wiper is acquired in step 56. Each image is processed using two one 
dimensional high pass filters; one for the vertical direction and one for the 

10 horizontal direction as indicated in step 58. Since the windshield wiper will appear 
as a vertical line in an image, there should be a significantly higher high frequency 
component in a horizontal direction than in a vertical direction. Thus, a vertical 
high pass filter may be implemented using a 3 x 3 matrix described above with the 
coefficient E set to 2, coefficients B and H set to -1 and the rest set to zero. A 

15 horizontal filter is implemented with the coefficient E set to 2, the coefficients D 
and F set to -1 and. the rest set to zero. The sum of each component is tallied in 
the same fashion as used to compute the Laplacian discussed above. In step 60, 
the ratio of the horizontal component to the vertical component is computed. If the 
horizontal component is much greater than the vertical component, a vertical line 

20 is assumed to be present in the image indicating the presence of the windshield 
wipers. 

If the wipers of the automobile are designed in such a way that the 
wipers are never approximately vertical when it crosses the rain sensing area, the 
filters described above can be modified to accommodate such a configuration. For 
25 example, various other edge detection methods well known in the art of image 
processing can also be used. Additionally, if the wiper speed for the vehicle 
windshield wipers is so fast that it blurs slightly in the image for the necessary 
exposure time, the horizontal filter can be modified to subtract the pixels two 
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positions to the left and right of the current pixel instead of the pixels immediately 
next to the current pixel. 

After the wiper has cleared the image sensing area, as indicated in 
step 62, another image of the windshield is acquired in step 64 for which the 
5 Laplacian is computed. This computation is used as a zero point measurement that 
may be subtracted from all subsequent measurements until the next wipe. In this 
way, long term high frequency spatial components in the image of dirty 
windshield, cracks, scratches and frozen ice will not contribute to the detected 
amount of rain. 

10 If the windshield wiper is not detected within a given time frame, 

the system assumes that a malfunction has occurred, which can be caused as a 
result of the windshield wiper being frozen to the windshield. During such a 
condition, the operation of the moisture sensor in accordance with the present 
invention can be suspended for a period of time to allow the ice to thaw. If outside 

15 temperature information is available, freezing climate conditions can be taken into 
account to decide if the wipers are failing because of a mechanical malfunction or 
due to ice. 

The system is also able to adapt to varying light levels. In 
particular, during selected cycles, the average gray scale value of the image may 

20 be computed. If this value is high, indicating an over exposure to light, the image 
sensor integration time in the following cycle may be reduced to lower the average 
brightness. Similarly, if the light level is low, the integration time may be 
increased. In relatively dark conditions, some image sensors may not be capable 
of collecting enough light in a reasonable time to adequately image moisture such 

25 as raindrops. In such a situation, an additional illuminator may be provided to 
briefly illuminate the area of interest from behind while the image is being taken. 
If the windshield of the vehicle is not highly absorbent to infrared radiation, a near 
infrared illuminator can be used as long as the wavelengths are within the 
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detectable region of the image sensor. An infrared illuminator has the benefit of 
not being visible to the human eye and thus not distracting to the driver. 

Fog Detector 

In order to eliminate many of the operational deficiencies of known 
5 moisture sensing systems, the system in accordance with an alternate embodiment 
of the invention includes a system for detecting fog on the interior and the exterior 
of the windshield. As illustrated in FIGs. 2 and 3, a light emitting diode (LED) 
may be used for detecting fog both on the interior and the exterior of a vehicle 
windshield 26. Two different embodiments of the fog detection systems are 

10 disclosed both of which rely on the difference in the way the light from the LED 
66 is reflected from the windshield 26 in the presence of fog. Fog detection may 
be done in alternate processing cycles with the rain detection. In particular, many 
cycles may be used for rain and moisture detection between fog detection cycles 
due to the slow onset of windshield fogging. At the beginning of the fog detection 

15 cycles, an image may be acquired using a window containing the expected location 
of the spots. 

In a first embodiment, a light source that is either highly collimated 
or focused to a point at the windshield distance can be used. The light source can 
be either infrared emitting or visible emitting depending upon the absorption 

20 characteristics of the windshield. Infrared sources are preferable since they are not 
visible to the human eye and therefore do not pose a distraction. An infrared LED 
66 may be used along with a lens 68 of a focal length equal to the windshield 
distance as generally shown in FIG. 2. The LED 66 may be positioned a few 
millimeters above the main optical assembly and angled so that the projected light 

25 as shown in FIG. 3 is aimed at the position of the main optical axis on the 
windshield 26. The LED 66 is initially turned off and an image is taken. 
Immediately following, the LED is turned on and a second image is taken. The 
difference between these images is used for spot detection. If no fog is present, 
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the light will reflect from the windshield 26 at the Snell angle which will carry it 
well outside the field of view of the image sensor 32. If fog is present, the light 
will be reflected by Lambertian reflectance causing the fog to be imaged as a small 
spot 70 (FIG. 3). Due to the angles of the windshield 26 and the light source 66, 
5 outside fog will produce a spot 70 lower than a spot 72 produced by fog in the 
interior surface of the windshield 26. The location of these spots 70, 72 may be 
used to represent the presence of interior and/or exterior fog on the windshield 26. 
If fog is present on the interior of the windshield, exterior fog detection is not 
possible. However, this limitation is of no consequence because vision would be 
10 impaired anyway. The following truth table indicates the conclusions drawn from 
the presence of each spot: 



Exterior (Lower) 
Reflected Spot 


Interior (Upper) 
Reflected Spot 


Result 


Present 




Exterior Fog 




Present 


Interior Fog 


Not Present 


Not Present 


No Fog 



In an alternate embodiment an infrared or, if necessary, a visible 
LED is used. The LED must be either relatively small or be used in conjunction 
with a pin hole and aimed such that the light from the LED reflects off the 

20 windshield 26 and on to an image sensor at the Snell angle. In such a 
configuration, two reflections will occur: one off the inside of the windshield and 
one from the outer surface of the windshield by specular reflection. In this 
embodiment if fog is not present, the spots are reflected at the Snell angle and are 
visible by the image sensor. If there is fog on the exterior surface of the 

25 windshield 26, the spot from the interior reflection will be present but the spot 
from the exterior refection will be a blur. The difference image is analyzed for 
interior fog. Initially a 3 x 3 Laplacian filter is used to create a new image which 
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contains only the high frequency components of the original image. In this way 
the blurred reflection from fog is illuminated. The spots are detected by taking the 
maximum value of a group of pixels in the expected region of each spot. A group 
slightly larger than the expected size of the spot may be used in order to correct 
5 for slight misalignments and for the fact that the Laplacian will only preserve the 
edges of the spots where the high frequency components are present. Also slight 
fog may cause the spot to grow but may not require action. For each spot, if the 
value is reasonably greater than zero, it is determined to be present. The following 
truth table indicates the detection of fog for the alternate embodiment. 



Exterior (Upper) 
Reflected Spot 


Interior (Lower) 
Reflected Spot 


Result 


Present 


Present 


No Fog 


Not Present 


Present 


Exterior Fog 


Not Present 


Not Present 


Interior Fog 



15 If the exterior fog is detected, the windshield wipers may be 

actuated to help remove the fog or if desired a warning light can be used to indicate 
this condition to the driver. Interior fog detection can be used to automatically 
actuate the vehicle defrost or defogger system preventing the driver from waiting 
until the significant fog has developed. More involved processing methods may 

20 include determining the size of the spot by computing the distance between the 
horizontal edges of the spot and thus having a measurement of the quantity of fog 
on the windshield which can be compared to a threshold. 

Obviously, many modifications and variations of the present 
invention are possible in light of the above teachings. Thus, it is to be understood 

25 that, within the scope of the appended claims, the invention may be practiced 
otherwise than as specificlly described above. 
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What Is Claimed fa 

1 . A system for detecting moisture on a surface comprising: 
an image sensor; 

an optical system operable to image a portion of the surface onto the 
5 image sensor, to focus objects at the surface onto the image sensor, and to blur 
objects away from the surface; and 

a processor in communication with the image sensor, the processor 
operative to determine spatial composition of the image whereby a signal may be 
generated based on the determined spatial composition. 

10 2. The system for detecting moisture on a surface as in claim 1 , the 

surface comprising a vehicle window. 

3 . The system for detecting moisture on a surface as in claim 1 , the 
system farther comprising an electrochromic window disposed between the surface 
and the imaging optical system, the electrochromic window operative to reduce the 

15 amount of light imaged by the image sensor. 

4. The system for detecting moisture on a surface as in claim 1 , the 
image sensor comprising a CMOS active pixel sensor. 

5 . The system for detecting moisture on a surface as in claim 1 , the 
image sensor comprised of a plurality of pixel sensors, each pixel sensor operative 

20 to produce an analog signal indicative of a pixel illumination level, the system 
further comprising an analog-to-digital converter for determining a gray scale 
value from the illumination level. 



25 



6 . The system for detecting moisture on a surface as in claim 5 , the 
image sensor having a variable integration time, the system operable to control the 
variable integration time of the image sensor. 
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7. The system for detecting moisture on a surface as in claim 5, the 
processor operable to determine image spatial composition by high pass digital 
filtering of a set of gray scale values. 

8. The system for detecting moisture on a surface as in claim 7, 
5 wherein high pass digital filtering is accomplished using a Laplacian filter. 

9. The system for detecting moisture on a surface as in claim 1 , the 
system further including an illuminator to illuminate the imaged portion of the 
window. 

10. The system for detecting moisture on a surface as in claim 1, 
10 the vehicle having a wiper for wiping the surface along a path, the wiper driven 

by a wiper motor, the system further comprising a wiper motor control in 
communication with the processor and the wiper motor, wherein the generated 
signal is based on the determined spatial composition used to control the wiper. 

1 1 . The system for detecting moisture on a surface as in claim 10 
15 wherein the imaged portion of the surface includes the path of the wiper and the 

processor is further operable to determine if the wiper is in the imaged portion of 
the surface using the determined spatial composition. 

12. The system for detecting moisture on a surface as in claim 11 
wherein the processor is further operative to (a) acquire an image of the surface 

20 after the wiper has cleared the imaged portion of the surface; and (b) to determine 
a zero point using the image acquired after the wiper has cleared the imaged 
portion of the surface. 

13. The system for detecting moisture on a surface as in claim 1 1 
wherein the processor is further operative to suspend operation of the wiper if the 

25 wiper has not been detected within a time frame. 



WO 99/14088 



PCT/US98/18899 



-22- 

14. The system for detecting moisture on a surface as in claim 10, 
the system further comprising a driver on/off sensitivity control in communication 
with the processor. 

15. The system for detecting moisture on a surface as in claim 1, 
5 the system further comprising a light source in communication with the processor 

for projecting a light spot at the imaged portion of the surface in communication 
with the processor, the processor further operable to 
turn on the light source, 

obtain a first image of the surface with the light source on, 
10 turn off the light source, 

obtain a second image of the surface with the light source off, and 
determine if moisture is present on the surface using the first image 
and the second image. 

16. The system for detecting moisture on a surface as in claim 15 
15 wherein the moisture comprises fog or frost. 

17. The system for detecting moisture on a surface as in claim 15 
wherein the light source is positioned such that, when moisture is present on the 
surface, the light spot is reflected into the image sensor field of view. 

18. The system for detecting moisture on a surface as in claim 15 
20 wherein the light source is positioned such that, when moisture is not present on 

the surface, the light spot is reflected into the image sensor field of view. 

19. The system for detecting moisture on a surface as in claim 15 
wherein the surface comprises a window, the window having an interior side and 
an exterior side, the light source positioned such that the light spot reflected into 

25 the image sensor field of view from the interior side is located at a different 
position on the image sensor than the light spot reflected from the exterior side, 
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the processor further operative to determine the location of moisture on either the 
interior side or the exterior side from the position of the reflected light spot in the 
image sensor output. 

20. The system for detecting moisture on a surface as in claim 15 
5 wherein the processor is further operative to create a filtered image containing high 

frequency components of the image sensor output. 

21 . The system for detecting moisture on a surface as in claim 15 
wherein the light source comprises a visible light emitting diode. 

22. The system for detecting moisture on a surface as in claim 15 
10 wherein the light source comprises an infrared light emitting diode. 

23. The system for detecting moisture on a surface as in claim 15, 
the vehicle having a means for clearing frost or fog from the surface, the system 
further comprising a control in communication with the processor and the clearing 
means, the processor further operative to control moisture clearing based on the 

15 determined presence of moisture on the surface. 

24. A system for detecting moisture on a window, the window 
having an interior surface and an exterior surface, the system comprising: 

a light source projecting a light spot onto the window; 
an image sensor; 

20 an imaging optical system operable to image a portion of the 

window including the projected light spot onto the image sensor, to focus objects 
on the window onto the image sensor, and to blur objects away from the window, 
wherein 

the light source is positioned such that the light spot reflected into 
25 the image sensor field of view from the interior surface is located at a different 
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position on the image sensor than the light spot reflected from the exterior surface; 
and 

a processor in communication with the image sensor and the light 
source, the processor operative to determine the presence of moisture and to 
5 determine the location of moisture on either the interior surface or the exterior 
surface, whereby a signal may be generated based on the determined presence of 
moisture and the determined location of moisture. 

25 . The system for detecting moisture on a window as in claim 24, 
the light source positioned relative to the window and the imaging optical system 

10 such that, when moisture is present on at least one of the interior surface and the 
exterior surface, the light spot is reflected from the window, through the imaging 
optical system, and onto the image sensor. 

26. The system for detecting moisture on a window as in claim 24, 
the light source positioned relative to the window and the imaging optical system 

15 such that, when moisture is present on at least one of the interior surface and the 
exterior surface, the light spot is not reflected onto the image sensor. 

27. The system for detecting moisture on a window as in claim 24 
wherein the light source comprises a visible light emitting diode. 

28. The system for detecting moisture on a window as in claim 24 
20 wherein the light source comprises an infrared light emitting diode. 

29. The system for detecting moisture on a window as in claim 24, 
the vehicle having a means for clearing fog or frost from the window, the system 
further comprising a control in communication with the processor and the clearing 
means, the processor further operative to control the clearing means based on the 

25 determined presence of moisture on the window. 
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30. The system for detecting moisture on a window as in claim 24, 
the processor further operable to extract high spatial frequency components of the 
image sensor output. 

31. A method for detecting moisture on a surface comprising: 

5 acquiring an image of a portion of the surface, the image having 

objects on the surface in focus and objects away from the surface blurred; 

generating a signal corresponding to the acquired image; 
determining spatial high frequency components of the signal; and 
determining the moisture using the spatial high frequency 

10 components. 

32. The method for detecting moisture on a surface as in claim 3 1 
wherein generating the signal comprises digitizing the acquired image. 

33. The method for detecting moisture on a surface as in claim 31 
further comprising: 

15 determining if a radiative loading greater than a threshold is imaged; 

and 

temporarily reducing the radiative loading greater than the threshold 
from the image. 



34. The method for detecting moisture on a surface as in claim 31 
20 further comprising: 

determining the amount of light integrated by an image sensor 
having a variable light integration time; and 

controlling the variable integration time of the image sensor based 
on the determined amount of integrated light. 
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35. The method for detecting moisture on a surface as in claim 31 
wherein determining spatial high frequency components of the digitized signal 
comprises high pass digital filtering of the digitized signal. 

36. The method for determining moisture on a surface as in claim 
5 31 wherein the high pass digital filtering is accomplished using Laplacian filtering . 

37. The method for detecting moisture on a surface as in claim 31 
further comprising illuminating the imaged portion of the surface. 

38. The method for detecting moisture on a surface as in claim 3 1 , 
the vehicle including a wiper for wiping the surface along a path, the imaged 

10 portion of the surface including the path of the wiper wherein the method further 
comprises detecting the presence of the wiper in the imaged portion of the surface. 

39. The method for detecting moisture on a surface as in claim 38 
wherein detecting the presence of the wiper comprises determining if a plurality 
of high frequency components in the direction of wiper travel through the imaged 

15 portion of the surface are greater than a plurality of high frequency components 
in a direction normal to the direction of wiper travel through the imaged portion 
of the surface. 

40. The method for detecting moisture on a surface as in claim 39 
further comprising controlling the rate of the wiper based on the determined signal 

20 representative of moisture. 

41 . The method for detecting moisture on a surface as in claim 40 
further comprising preventing further operation of the wiper if the wiper has not 
been detected within a time frame. 
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42. A method for detecting moisture on a surface, the vehicle 
having at least one wiper for wiping the surface, the vehicle including an image 
sensor operable to image the surface in a region covered by the at least one wiper, 
the method comprising: 

5 acquiring an image of the surface, the image having objects at the 

surface in focus and objects away from the surface blurred; 

determining a sum, the sum comprising a Laplacian for each pixel; 
determining a difference of a zero calibration subtracted from the 

sum; 

10 repeating the steps of acquiring the image , determining the sum, and 

determining the difference if the difference is less than a threshold, otherwise 
actuating at least one wiper; 

determining that the at least one actuated wiper has wiped the 
imaged region of the surface; and 
15 determining as the zero calibration the Laplacian of an image taken 

after the imaged region of the surface has been wiped. 

43. The method for detecting moisture on a surface as in claim 42 
wherein determining that the at least one wiper has wiped the imaged region of the 
surface comprises: 

20 acquiring an image of the surface, the image having objects at the 

surface in focus and objects away from the surface blurred; 

determining a first filtered image by passing the image of the 
surface through a first one-dimensional high pass filter, the one dimension in the 
direction of wiper travel through the image of the surface; 

25 determining a second filtered image by passing the image of the 

surface through a second one-dimensional high pass filter, the one dimension in 
the direction normal to the direction of wiper travel through the image of the 
surface; 
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repeating the steps of acquiring an image of the surface, 
determining a first filtered image, and determining a second filtered image if a first 
sum of values of the first filtered image is not greater than a second sum of values 
of the second filtered image, otherwise 
5 delaying for a time to allow the wiper to pass out of the image of 

the surface. 
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